You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gpu-config [10/13]: add the GPU configuration manager main loop - #3256
The changes add a GPU configuration runtime. It parses and matches configuration documents, checks driver and workload evidence, applies power, clock, and ECC settings through NVML, and records outcomes in a locked state file. Kubernetes clients read nodes, pods, DRA resources, and ConfigMaps. A gate and applier coordinate decisions for driver loads, while status publishing updates node labels for sync state, schema version, and pending ECC activation.
Merge Risk: Moderate · up to f9ca8
This adds the GPU configuration manager's per-node loop and its supporting decision, parsing and gating logic. It is not yet wired into the operator, but several logic gaps should be fixed before it ships. A failed init container can permanently skip configuration for a driver load. Certain merged YAML entries can bypass validation and apply the wrong settings. The startup gate can release early after a brief module observation. Address these before merging.
Security architecture risk: Minimal · up to f9ca8
The inspected change establishes no material new security or failure-containment risk. GPU changes are guarded by workload and driver checks, and failed applications do not record success or release an expected startup gate. The new runtime is not activated by this change.
Retained concerns
No architecture-level concerns identified.
Security review details
Security Blast Radius
inferred — When activated, the privileged setting path operates across the configured node’s enumerated GPUs, not merely one workload’s allocation. A shared document can therefore influence multiple consuming nodes. Current production reachability and deployment-granted authority are not established by this staged addition.
Trust Boundaries and Controls
observed — The configuration-to-GPU transition is guarded by allocator-release evidence, workload checks, driver readiness, load-ID revalidation after device handles open, and range checking before any setting write. Workload exemptions are scoped to the configured operator namespace rather than arbitrary tenant labels.
Resilience and Maintainability Implications
observed — Failure containment separates hardware application from administrative status publication. Application failures leave no successful decision for an expected gate to accept. Explicit gate bypasses are recorded as evidence that prevents a later unsafe application; failed status publication queues a retry without reapplying settings.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
Other comments: 2
internal/gpuconfig/gate/gate.go: 1
185-187: Minor · Functional Correctness · Quick win
Reset the absence timer when the locked recheck sees a module.
After NoModuleBound expires, the first LoadID call can report absence while this recheck reports presence. This branch leaves moduleAbsent true. If the module disappears before the next poll, the gate immediately records module zero instead of waiting for a new absence interval. A later applier then skips configuration.
Set g.moduleAbsent = false before returning here. This preserves the existing timeout for continuous absence and restarts it after an observed load.
internal/gpuconfig/decision/decide_test.go: 1
669-673: Minor · Functional Correctness · Quick win
Reject unexpected errors in successful test cases.
If a successful case returns Result{} and an error, this branch accepts None, unchanged state and closed NVML. It then returns before checking the expected outcome. For example, the settings-application case can pass without applying settings.
Require NoError when neither wantErrIs nor wantFatal expects an error. As per path instructions, flag assertions “that would still pass if the behavior under test were broken.”
These assertions would pass if DeviceByIndex(0) returned nil, nil. TestDevice does not catch that regression because it constructs device directly.
Give the mock device a known UUID. Retain the returned device and check its UUID through apply.Device. This detects a missing or incorrect handle wrapper.
As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”
This test calls DocumentChanged but leaves staticDocs unchanged. It checks only node-read counts. It would pass if publishing continued to use an obsolete document snapshot.
Start with an applied document and InSync. Replace the document, send the notification, and assert OutOfSync after the jitter. Also assert that settings are not applied again. As per path instructions, flag tests where “the test only checks that a mock was called.”
Reviewing files that changed from the base of the PR and between 246a9ca and f9ca861.
Files ignored due to path filters: 39
go.sum is excluded by !**/*.sum, !go.sum
vendor/github.com/NVIDIA/go-nvml/LICENSE is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_linux.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_other.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/api.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers.h is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers_static.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const_static.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/device.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/doc.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/dynamicLibrary_mock.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/event_set.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/gpm.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/init.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/lib.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/memory_limits.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/README.md is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/computeinstance.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/device.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/eventset.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/extendedinterface.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpmsample.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpuinstance.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/interface.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/unit.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgpuinstance.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgputypeid.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.h is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/refcount.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/return.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/system.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/types_gen.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/unit.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/vgpu.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/zz_generated.api.go is excluded by !vendor/**, !**/zz_generated.*.go
vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 36
go.mod
internal/gpuconfig/applier/applier.go
internal/gpuconfig/applier/applier_test.go
internal/gpuconfig/apply/apply.go
internal/gpuconfig/apply/apply_test.go
internal/gpuconfig/apply/fake/fake.go
internal/gpuconfig/apply/nvml.go
internal/gpuconfig/apply/nvmllib/nvmllib.go
internal/gpuconfig/apply/nvmllib/nvmllib_test.go
internal/gpuconfig/config/document.go
internal/gpuconfig/config/document_test.go
internal/gpuconfig/config/match.go
internal/gpuconfig/config/match_test.go
internal/gpuconfig/config/schema.go
internal/gpuconfig/decision/decide.go
internal/gpuconfig/decision/decide_test.go
internal/gpuconfig/decision/evidence.go
internal/gpuconfig/decision/helpers_test.go
internal/gpuconfig/decision/pods.go
internal/gpuconfig/gate/gate.go
internal/gpuconfig/gate/gate_test.go
internal/gpuconfig/kube/client.go
internal/gpuconfig/kube/client_test.go
internal/gpuconfig/kube/documents.go
internal/gpuconfig/kube/documents_test.go
internal/gpuconfig/loadid/driver.go
internal/gpuconfig/loadid/loadid.go
internal/gpuconfig/loadid/loadid_test.go
internal/gpuconfig/publish/publish.go
internal/gpuconfig/publish/publish_test.go
internal/gpuconfig/statefile/file.go
internal/gpuconfig/statefile/file_test.go
internal/gpuconfig/statefile/state.go
internal/gpuconfig/statefile/state_test.go
internal/gpuconfig/status/sync.go
internal/gpuconfig/status/sync_test.go
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 2 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Quick win
Preserve merged keys during entry-name validation.
If gpus contains <<: {1: entryA, "1": entryB}, strict YAML decoding accepts the distinct key types. JSON conversion then collapses both names to "1". In yaml/v2 v2.4.4, MapSlice decoding discards merged items, so walkKeys cannot reject the integer entry name. The document can therefore apply one entry's settings instead of failing validation. The dependency implementations establish this path. (raw.githubusercontent.com)
Use a key-preserving representation that includes merged mappings. Add a regression case with these two names and different settings. Keep the existing valid merge case supported.
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Quick win
Require successful termination before recording a gate pass.
If toolkit-validation or gpu-config terminates with ExitCode=1, these checks still report that the allocator wait or gate passed. Decide then records Skipped. Later decisions return Idle, even if the failed container recovers.
Require ExitCode == 0 for these termination checks. Keep the later-container checks, which can independently prove progress.
henry118
changed the title
gpu-config: add the GPU configuration manager main loop
gpu-config [10/13]: add the GPU configuration manager main loop
Nov 19, 2026
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3246 (10/13). Depends on: #3247 to #3255. Review only the last commit.
Adds
internal/gpuconfig/applier: the manager's per-node loop that binds toa driver load, decides once, and publishes status. Not used yet.